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Letter from the Editor 


Summer brings time to reflect and recharge. The Summer 2017 volume of AIR Professional Files 
presents four articles with intriguing ideas to consider as you plan for the next academic year. | 


Data governance is a pressing issue for many IR professionals, as sources of data proliferate and os 
Association for 


challenge our ability to control data integrity. In her article, Institutional Data Quality and the Institutional Research 
Data Integrity Team, McGuire synthesizes and interprets results from 172 respondents to an AIR- 
administered survey of postsecondary institutions on their data integrity efforts. She describes the current state of data governance 


and offers strategies to encourage institutional leaders to invest in data quality. 


Those of us who work in assessment often take it for granted that assessment results will be used for learning improvement. Fulcher, 
Smith, Sanchez, and Sanders challenge this assumption by analyzing information from program assessment reports at their own 
institution. Needle in a Haystack: Finding Learning Improvement in Assessment Reports uncovers many possible reasons for the gap 
between obtaining evidence of student learning and using that evidence for improvement. The authors suggest ways to promote 
learning improvement initiatives, and share a handy rubric for evaluating assessment progress. 


Institutional researchers are beset with requests to form peer groups, and it seems that no one is ever satisfied with the results. 

Two articles in this volume present very different methodologies for forming sets of comparison institutions. In her article, A Case 
Study to Examine Three Peer Grouping Methodologies, D‘Allegro compares peer sets generated by different selection indices. She 
offers guidance for applying each index and encourages cautious interpretation of results. Rather than rummaging around for 

the perfect peer set, Chatman proposes creating a clone, or doppelganger university, one that is constructed from disaggregated 
components drawn from diverse data sources. In Constructing a Peer Institution: A New Peer Methodology, he walks us through the 
process of creating peers for faculty salaries, instructional costs, and faculty productivity. While the constructed peer approach has its 
challenges, the appeal of achieving a perfect fit peer is undeniable. 


| hope your summer “reflection” inspires you to share your work with your IR colleagues through AIR Professional Files. 


Sincerely, 


Sharron L. Ronco 
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CONSTRUCTING A PEER INSTITUTION: A NEW 
PEER METHODOLOGY 


Steve Chatman 


About the Author 

Steve Chatman is an institutional 
research analyst in the Office of 
Institutional Research and Decision 
Support at the University of California, 
Merced. 


Abstract 

Whatever your method of selecting 
institutions for comparison and 
benchmarking, you can both increase 
the validity and accuracy of those 
comparisons and extend the value 

of comparisons to department and 
college levels by constructing a 

peer institution from disaggregated 
components. This paper will 
demonstrate the methodology using 
the National Study of Instructional 
Costs and Productivity (Delaware Cost 
Study), the Faculty Salary Survey by 
Discipline (Oklahoma State University 
[OSU]), and Academic Analytics, LLC, 
to construct better peer institutions 
with comparative statistics at campus, 
college, and department levels for 
faculty salaries, instructional costs, and 
research activity. The methodology can 
also be used to fine-tune traditional 
peer methodologies and should be 
added to the institutional research 
arsenal of cluster-, threshold-, hybrid-, 
and panel-based peers. 


NARRATIVE 


In the most influential institutional 
research document describing peer 
institution selection, Paul Brinkman 
and Deb Teeter (1987, p. 7) wrote, “In 
developing peer groups, it is unrealistic 
to expect to find perfect matches, 
‘clones’ as it were, for the home 
institution.” In fact, practitioners soon 
discover that the use of even a handful 
of narrowly described thresholds 
(same schools and colleges of same 
relative sizes) will eliminate all other 
universities, and the researcher is left 
with an off-the-rack fit instead of a 
tailored fit. This paper asserts that 
Brinkman and Teeter were wrong 
about finding perfect matches. There 
is an alternative that will produce a 
near-perfect match: that is, a clone or 
doppelganger university. It just will 
not be a brick-and-mortar university. 
In fact, it won't exist except on 
spreadsheets or in computer code. 


Traditional methods of peer group 
selection can be classified into 
developed or predetermined types. 
These types are not mutually exclusive 
and most peer selection processes 
incorporate elements of multiple 
types. Predetermined types are easily 
communicated publicly and include 
the following: 


1. Natural peers are based on 
geography, athletics conferences, 
consortiums, or similar factors. 
These peers are particularly 
useful when communicating 
with legislators or the public in 
general. 

2. Traditional peers are based on 
long-term associations or rivalries 
(e.g., lvy League, State versus 
University of). 

3. Jurisdictional peers are based on 
political, legal, and administrative 
systems (e.g., state, regional, 
campuses of the university 
system, accreditation regions). 

4. Classification-based peers are 
most often based on Carnegie 
Basic Classification or a subset 
thereof. 


Developed peers rely on measured 
characteristics and can vary from 
simple (e.g., disciplinary composition 
clusters, public Research 1 and 2 [R1 
and R2]), to complex (e.g., student 
characteristics, funding levels, 
composition by student levels, 
professional programs), and include the 
following: 


1. Cluster analysis is more 
statistically complex. It sorts 
institutions into groups based 
on composition dimensions. 
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Table 1. Home U Instruction by Department and College Expenditures Compared to Expenditures at National Research 
Universities (Data Are Fictitious) 


Home U Degree CIP Delaware Home U FTE Home U Home U 


Programs / Majors Discipline if Students Instruction Instruction $ / 

Different (Ugrad SCH / 15 Expenditure FTE Student 
mCi y-(o~) 015 Wi 4) 

Anthropology 45.02 127 $888,679 $6,975 

Cognitive Sciences 30.25 42.00 Psychology 208 $1,508,545 $7,269 

Economics 45.06 229 $1,100,499 $4,810 

History 54.01 114 $1,035,698 $9,078 

Literatures and 16.01 458 $3,719,811 $8,125 

Cultures 

Management 52.02 115 $565,035 $4,928 

Political Science 45.10 173 $1,721,097 $9,968 

Psychology 42.01 827 $3,734,230 $4,517 

Sociology 45.11 273 $1,236,805 $4,529 

School of Social 2523 $15,510,399 $6,148 

Sciences, Arts, and 

Humanities 

Applied Mathematics 27.03 27.00 Mathematics 782 $3,300,100 $4,218 
and Statistics 

Bioengineering 14.05 40 $805,709 $19,943 

Biological Sciences 26.01 605 $3,392,147 $5,611 

Chemistry 40.05 492 $2,905,605 $5,905 

Earth Systems 40.06 104 $1,607,946 $15,506 

Sciences 

Physics 40.08 219 $1,941,943 $8,863 

School of Natural 2242 $13,953,450 $6,223 

Sciences 

Computer Science and | 14.09 11.07 Computer 223 $2,474,021 $11,083 

Engineering Science 

Environmental 14.14 14.08 Civil 113 $1,632,681 $14,498 

Engineering Engineering 

Materials Science and 14.18 77 $844,570 $10,921 

Engineering 

Mechanical 14.19 124 $2,047,071 $16,529 

Engineering 

School of Engineering 537.0 $6,998,343 $13,032 

Writing Program 23.13 725.2 $4,340,547 $5,985 

Home U Overall 6,027.3 $40,802,739 $6,770 
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Delware Cost Study 


Home U Instruction 


Home U - Delaware 


'feVfolalitate mn Elareyar-) 


$ Difference Times 


Instruction $ Per $ Per Student / Instruction $ Per Instruction Home U FTE 
FTE Student National Research Student Expenditure by Students 
Univ $ Per Student Home U FTES 
$5,865 119% $1,110 747,299 141,380 
$5,632 129% $1,637 1,168,828 339,717 
$5,930 81% -$1,120 1,356,784 -256,285 
$6,157 147% $2,921 702,411 333,287 
$5,762 141% $2,363 2,638,036 1,081,775 
$6,948 71% -$2,020 796,704 -231,669 
$6,809 146% $3,159 1,175,687 545,410 
$5,632 80% -$1,115 4,656,162 -921,932 
$5,111 89% -$582 1,395,644 -158,839 
$5,802 106% $346 $14,637,554 872,845 
$5,172 82% -$954 4,046,918 -746,818 
$15,849 126% $4,094 640,300 165,409 
$6,824 82% -$1,213 4,125,677 -733,530 
$7,254 81% -$1,349 3,569,331 -663,726 
$9,531 163% $5,975 988,365 619,581 
$8,417 105% $446 1,844,165 97,778 
$6,785 92% -$563 $15,214,754 -1,261,304 
$10,175 109% $908 2,271,230 202,791 
$11,181 130% $3,317 1,259,167 373,514 
$15,508 70% -$4,587 1,199,285 -354,715 
$10,748 154% $5,781 1,331,140 715,931 
$11,286 115% $1,746 6,060,822 937,521 
$4,942 121% $1,043 3,583,938 756,609 
$6,553 103% $217 39,497,068 1,305,671 
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For example, institutions can be 
sorted based on relative mix of 
disciplinary degrees awarded. 

2. Threshold analysis is 
straightforward and 
easily communicated. The 
characteristics of potential 
peers have to fall within a 
range above and below the 
measured characteristic of the 
home institution. For example, 
if headcount enrollment at the 
home institution is 20,000, then 
peers would have enrollments 
between 17,500 and 22,500. 
Thresholds can be similarly 
applied to full-time equivalent 
(FTE) enrollment, admissions 
scores, in-state enrollment 
percentage, or almost anything 
commonly measured. 

3. Panel analysis relies on the 
expertise of professionals, 
typically institutional executives, 
who either nominate potential 
peers or eliminate potential peers 
identified by other methods. 

4. It is more common for the 
methodology to be a hybrid of 
other types in various sequences 
(e.g., cluster analysis followed 
by threshold analysis and then 
submission to a panel). 


In contrast with developed or 
predetermined institutional peers, 

the constructed peer methodology 
described in this paper is typically 
built from departmental or disciplinary 
components. Unlike institutional peers, 
the constructed peer methodology 
can use disciplinary components that 
vary from one department or school 

to another. Psychology might select 
Psychology peers and Biology might 
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select a different set of Biology peers. 
But even when the home institution is 
constrained to compare with a given 
institutional set, the constructed 

peer methodology can be based 

on the elemental characteristics of 
those institutions. Because the result 
is constructed from disciplinary 
components, the result will be useful 
at the level of the department and will 
be more accurate when aggregated to 
college and institutional levels. 


In spite of the availability of data 

to support a constructed peer 
methodology by department, 
especially for faculty salaries and 
disciplinary expenditures, the 
methodology has not contributed 

to the discussions of peer institution 
groups that were popular in the 
1980s and that continue to dominate 
institutional research practice: various 
cluster analysis techniques and some 
measure of judgment (panel, hybrid, 
threshold, panel) about institutional 
key or performance statistics (Brinkman 
&Teeter, 1987; Terenzini, Hartmark, 
Lorang, & Shirley, 1980; Trainer, 2008; 
Xu, 2008). There are two very good 
reasons to revisit peer methodology. 
First, good disaggregated data are 
available for critically important 
institutional research elements 
including faculty salaries (e.g., OSU 
since 1974), instructional costs and 
productivity (Delaware since 1992), and 
faculty research activities (Academic 
Analytics, LLC). Second, disciplinary 
composition should always be an 
institutional research consideration 
because it dramatically affects every 
aspect of teaching, research, and 
service; and every aspect of the student 
experience. There is less variance 


among universities by program than 
among programs within a university 
(Chatman, 2010). 


METHODOLOGY 


Information from the Delaware 

Cost Study, the OSU Faculty Salary 
Survey by Discipline, and Academic 
Analytics, LLC, will be used to construct 
doppelganger universities with 
comparative statistics at campus, 
college, and department levels for 
faculty salaries (OSU), instructional 
cost (Delaware), and faculty research 
and scholarly activity (Academic 
Analytics, LLC). The central feature 

of the methodology is constructing 

a peer by weighting comparative 

per capita or mean values to reflect 
the home institution composition. 

The methodology will be introduced 
using per capita instructional costs 
from the Delaware Cost Study. The 
other applications are similar in that 
they find a comparator per capita 
figures at the lowest available level 

of aggregation and weight that per 
capita figures using home institution 
amounts to create constructed 

or doppelganger departments. 

The resulting departments can be 
combined with others to produce a 
constructed peer or doppelganger 
university. The data shown are fictitious 
but generally reflect the characteristics 
of the University of California, Merced, 
a university that grew from farmland 
to research university in 10 years and 
continues to grow at a very rapid rate. 
The nearly 7,000 undergraduates in 
2016 had Hispanic, Pell Grant recipient, 
and first-generation majorities. 


Table 2. Department and College Level Faculty Salary Comparisons Using School of Natural Sciences at Home U and OSU 
Research Universities Average Salaries (2012-2013) 


Home U (Actual) 


Ladder Rank | Content Area Four-Digit OSU R1 & Home U Comparator- Home 
CIP Code R2 Expenditure | Based U/ OSU 
Expenditure R1&R2 
Professor Applied Mathematics 122,866 
Assoc. Prof. | Applied Mathematics 82,000 83,941 328,000 335,764 98% 
Asst. Prof. Applied Mathematics 77,200 73,884 308,800 295,536 104% 
Professor Biology, General 142,400 126,463 427,200 379,389 113% 
Assoc. Prof. Biology, General 83,717 84,375 502,302 506,250 99% 
Asst. Prof. Biology, General 74,040 72,848 740,400 728,480 102% 
Professor Biomedical/Medical 149,400 155,250 149,400 155,250 96% 
Engineering 
Assoc. Prof. Biomedical/Medical 99,300 104,157 99,300 104,157 95% 
Engineering 
Asst. Prof. Biomedical/Medical 89,400 83,843 178,800 167,686 107% 
Engineering 
Professor Chemistry 117,667 135,046 353,001 405,138 87% 
Assoc. Prof. Chemistry 88,650 84,958 177,300 169,916 104% 
Asst. Prof. Chemistry 74,667 74,369 448,002 446,214 100% 
Professor Ecology, Evolution, 109,350 128,697 218,700 257,394 85% 


Systematics, and 
Population Biology 


Assoc. Prof. Ecology, Evolution, 82,500 91,106 82,500 91,106 91% 
Systematics, and 
Population Biology 


Asst. Prof. Ecology, Evolution, 78,750 77,694 315,000 310,776 101% 
Systematics, and 
Population Biology 


Professor Physics 40.08 151,700 | 1 122,345 151,700 122,345 124% 
Assoc. Prof. | Physics 40.08 85,425 | 4 84,901 341,700 339,604 101% 
Asst. Prof. Physics 40.08 78,960 |5 75,386 394,800 376,930 105% 
School of Natural Sciences 


Professor Overall 130,000 131,952 1,300,001 1,319,516 99% 


Assoc. Prof. Overall 85,061 85,933 1,531,102 1,546,797 99% 


Asst. Prof. Overall 76,961 75,020 2,385,802 2,325,622 103% 


5,216,905 
Mean Overall | 88,422 99,708 89% 
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Figure 1. Instruction Productivity and Cost Difference by Discipline Relative to Public Research Universities Nationwide (Data 


Are Fictitious) 


Figure 1: Instruction Productivity and Cost Difference by Discipline Relative to 
Public Research Universities Nationwide (Data Are Fictitious) 
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Comparing Instructional 
Costs at the Constructed Peer 
Institution 

Please note that the data here and 
elsewhere in the report are fictitious 
and are offered to illustrate the 
methodology. Steps 1 through 4 
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describe the methodology for one 
department, Sociology. The same steps 
apply to other disciplines/departments 
and the results can be aggregated to 
colleges or to the university total. 


1. The home institution instructional 


expenditure in Sociology was 
$1.2 million. 

2. The expenditure per FTE student 
(based on Sociology student 
credit hours [SCHs] by level) was 
$4,529 at the home campus. 

3. The per student expenditure in 


sociology for research universities 
(R1 and R2) from the Delaware 
Cost Study was $5,111, compared 
to $4,529 at the home campus. 
The home institution therefore 
spent 89% of the “expected” 
amount, or $582 less per student. 
4. The home institution had 273 
FTE students in Sociology and 
therefore spent about $159,000 
less to deliver sociology 
instruction than expected. 


Steps 1 through 4 were repeated 
for the other departments and 

then aggregated to the college 

and university levels in Table 1. For 
the School of Social Sciences, Arts, 
and Humanities, the instructional 
expenditure was 106% of the 
constructed peer; Engineering was 
115%; and Natural Sciences was 
92%. Overall, the home institution 
instructional expenditure was 103% 
of the constructed research university 
peer. The difference per FTE student 
overall was $217, or $1.3 million in 
total. 


In this example, all public research 
universities were used for comparison 
but Delaware supports analysis by 
selected peers and the peer set could 
even vary based on the department 

or college, especially if the home 
institution participates in a data- 
sharing consortium (e.g., Association of 
American Universities Data Exchange 
[AAUDE)). It is easy to imagine that an 
Engineering peer set could differ from a 
Natural Sciences peer set, etc. 


Table 1 shows the detail behind 
computing comparisons and the 
difference between the local university 


and the comparative figures per FTE 
student by department, college, and 
campus. Figure 1 arrays expenditure 
differences along two axes. The x-axis 
is the difference per FTE student and 
the y-axis is the difference for all FTE 
students (difference per student times 
number of FTE students). The two axes 
of Figure 1 are used because a big 
difference per FTE student in a small 
department can have less institutional 
impact than a small difference in a 
large department. 


In examining the scatterplot in Figure 
1, it is clear that the per student 
institutional composite was close to 
that for the constructed peer, but that 
there was a great deal of variation 

by department and school. If the 
analysis was limited to institution-level 
measures, the school and departmental 
differences would have been obscured. 
That is a danger of institution-level 
measures. The composite can be at the 
mean peer value, suggesting normative 
performance, but be made up of values 
showing wide variation. Funding 

at the institutional level without 
consideration of disciplinary patterns 
makes that misleading outcome more 
likely. The results by school show 

that one school, Natural Sciences, is 
spending less than expected for natural 
science disciplines and is helping 

to offset the other schools that are 
spending more than expected for 

their disciplines. Both schools (Natural 
Sciences; and Social Sciences, Arts, 

and Humanities) are actually spending 
very similar amounts per FTE student. 
However, the expected expenditure for 
natural sciences is $563 more per FTE 
student in this example. It is reasonable 
to expect the dean of Natural Sciences 


to make these differences known in the 
next budget cycle. Please recall that 
these are not actual amounts and are 
used to illustrate the methodology; 
even if accurate, however, the results 
are not intended to be prescriptive. 
They do not show programs to be cut 
or where investments are needed, but 
they do identify areas of greater or 
lesser spending than is average and 
raise the question of whether those 
spending differences were intentional 
or a historical artifact. 


Other Examples 

The technique is generally applicable. 
Any comparative measure from an 
outside source that is available at a low 
level of aggregation can be weighted 
to reflect local composition and 
thereby create more-accurate, more- 
valid, and more-useful statistics for the 
department, school, and university. 
The following will illustrate the 
methodology using faculty salaries and 
faculty professional research activity 
but it could be extended to almost any 
measure. 


Faculty Salary Comparison 

The predominant factors associated 
with variance in faculty salaries 

are discipline and rank. Unless the 
comparator peer set has the same 
faculty composition by rank and 
discipline, there will be error that 
might be masked at the campus 
level. That error can be controlled by 
constructing a peer that does have 
the same disciplines and ranks in 
the same amounts. The following 
example illustrates the methodology 
using OSU Faculty Salary Survey by 
Discipline averages for public R1 

and R2 institutions. As was the case 
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Table 3. Home U Data Compared to Public and Private University Faculty Academic Analytics for Natural Sciences (Data Are 


Fictitious) 


Natural 
Sciences 
Disciplines 


Applied 
Mathematics 


Quantitative and 
Systems Biology 


Chemistry and 
Chemical Biology 


Environmental 
Systems 


Physics 


School of Natural 
Sciences 


Academic Analytics (Per Capita) 


yXer-Yo (Vaal (ey Home U Books’ Journal Citations Grants a CT e-Vals Honors 
Program Tenured (2005- Articles (2010-2014) (2010-2014) Dollars Elare| 
Ligeti} FT ate Mm 010) «- AM 9) (720 E20 £9) ATE Awards 
yNer-Yo (Vaal (ey Track 2014)” (Lifetime) 
Analytics Count from 

PXer-Yo (Waal (es 

Analytics 

Records 
Home U 11 0.3 10.3 55.6 1.1 78,497 0.3 
Academic 0.2 9.2 101.6 1.5 180,000 0.9 
Analytics 


Actual Output as Percent of Comparative-Average Based Output in Discipline 


Home U 40 0.1 10.2 153.1 1.0 181,365 0.3 
Academic 0.2 12.5 180.4 1.3 340,000 0.4 
Analytics 


Actual Output as Percent of Comparative-Average Based Output in Discipline 


Home U 16 0.3 14.6 288.3 11 206,538 0.3 
Academic 0.2 15.5 330.2 1.8 330,000 1.1 
Analytics 


Home U 


Academic 
Analytics 


27 


O41 
0.2 


Actual Output as Percent of Comparative-Average Based Output in Discipline 


12.2 
10.9 


152.6 
142.5 


1.5 
1.4 


235,405 
190,000 


0.4 
0.5 


Actual Output as Percent of Comparative-Average Based Output in Discipline 


Home U 18 0.2 22.4 125.1 0.9 110,077 0.6 
Academic 0.3 16.8 200.0 1:2 150,000 0.7 
Analytics 


Home U 


107 


Actual Output as Percent of Comparative-Average Based Output in Discipline 


Academic 
Analytics 
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Actual Output as Percent of Comparative-Average Based Output in Discipline 


Journal Articles 


Academic Analytics (Weighted) 


Citations 


(CTr-Talm BXe)|F- 1a 


Key ate) cer Tare | 
Awards 


3.3 113.3 611 12.0 863,464 3.0 

2:2 101.2 1,118 16.5 1,980,000 9.9 

150% 112% 55% 13% 44% 30% 
4.0 406.0 6,123 41.2 7,254,594 10.0 
8.0 500.0 7,216 52.0 13,600,000 16.0 
50% 81% 85% 19% 53% 63% 
4.8 233.6 4,612 18.1 3,304,601 5.0 

3.2 248.0 5,283 28.8 5,280,000 17.6 
150% 94% 87% 63% 63% 28% 
2.7 328.1 4,120 40.0 6,355,939 11.1 
5.4 294.3 3,848 37.8 5,130,000 13.5 
50% 111% 107% 106% 124% 82% 
3.2 403.2 2,252 16.9 1,981,379 10.1 
5.4 302.4 3,600 21.6 2,700,000 12.6 
60% 133% 63% 78% 713% 80% 
18.0 1,484.2 17,718 128.2 19,759,977 39.1 
24.2 1,445.9 21,064 156.7 28,690,000 69.6 
15% 103% 84% 82% 69% 56% 
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Figure 2. Relative Performance in Natural Sciences: Journal Articles 
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Figure 2: Relative Performance in Natural Sciences: Journal Articles 
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The vertical axis position is at 100% or a ratio of 1.0 -- when the ratio of observed actual amount to the amount based on comparative average was 1.0. 


for instructional expenditures, the 
mean salaries for the comparators by 
discipline and rank are weighted by the 
local university composition and the 
total expenditures are used to create 
college and institutional comparisons. 
For this example, the methodology 

will be applied to the School of 

Natural Sciences and illustrated 

using Chemistry. As shown in Table 2, 
Chemistry professors are paid $135,046, 
on average, at R1 and R2 schools. The 
home institution had three professors. 
If the home department paid the three 
professors exactly the national mean, 
the home department would have 
spent $405,138. The home department 
actually paid professors $353,001, or 
87% of the average. For all departments 
in the School of Natural Sciences, the 
home school spent $1,300,001 on 
professor salaries. If every department 
in the school had paid the national 
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public R1 and R2 average to each 
professor, the school would have spent 
99% of the aggregated $1,319,516 
amount. 


The constructed peer methodology is 
especially useful at Home University 
(Home U), an 11-year-old public 
research university, because its mix 
by rank and discipline is atypical. 
Because it is a new university, Home 
U has a much higher proportion 

of assistant professors and a much 
lower proportion of professors than is 
typical. It also has more STEM faculty 
than is typical of a public university. 
The unweighted campus mean, not 
adjusted for the higher proportion 

of assistant professors and lower 
proportion of professors, would be 
well below a simple institutional-level 
comparator even though both the 
comparisons by rank and the weighed 


institutional mean were above the 
comparator averages. This is illustrated 
for Natural Sciences in Table 2. By 

rank, faculty salaries were at or close 
to the national average: professor 
salaries were 99% of average, associate 
professors were 99% of average, and 
assistant professors were 103% of 
average. However, the overall average 
for the home institution was 89% of 
the overall national average. A result 
based on component comparisons that 
is different from the overall comparison 
is an example of the Yule-Simpson 
effect, defined as a trend appearing 

in different groups of data that 
disappears or reverses when the data 
are aggregated. In this case, means 
were close to the average by rank but 
substantially lower overall. As was 

the case for instructional costs, large 
differences for a few faculty should not 
be cause for alarm, but substantially 


different patterns by discipline might 
be cause for discussion, or there 

might be a strategic plan to recruit 
substantially more-competitive faculty 
in one area or another. The results 

are not prescriptive but should be 
illuminating. 


Faculty Professional Activity 

The third example relies on data 

from Academic Analytics, LLC, a 
service that gathers federal grants, 
books, honorific awards, journal and 
conference publications, and citations 
for individual faculty and makes 
those data available to subscribing 
institutions. The data values shown 
here are fictitious but the measures 
shown are available from Academic 
Analytics and are used with permission. 
Because faculty are identified by 
disciplinary area and institution type by 
Academic Analytics, the mean values 
for all faculty in a disciplinary area can 
be used as a comparative standard 
(Table 3). To make the explanation 
less complicated, analysis will again 
be limited to the School of Natural 
Sciences. 


For example, and using the 
comparative subset of these pseudo 
value statistics in Physics, the 
comparative average values per 
faculty member in Physics were about 
0.3 books (2005-2014), 16.8 journal 
articles (2011-2014), 200 citations 
(2010-2014), 1.2 grants (2010-2014), 
$150,000 grant dollars (2010-2014), 
and 0.7 honors and awards (lifetime). 
Because the home department had 
18 faculty members, the comparative 
average—based outcome for the 18 
faculty members in Physics was 5.4 
books, 302 journal articles, 3,600 


citations, 21.6 grants, $2,700,000 grant 
dollars, and 12.6 honors and awards. 
Actual counts were compared to the 
comparative average—based outcomes 
and expressed as percentages (60% 

to 140% for this Physics example). 

The comparative average—based 
outcomes and observed amounts can 
be aggregated to school and campus 
levels and can be used to identify 
areas of relative strengths. Those 
relative amounts can be expressed 
graphically. For this example, the 
relative percentages for journal articles 
in Natural Sciences disciplines are 
shown as Figure 2. Again, comparison 
at the school level (103%) obscures 

a substantial range by department 
(133% in Physics to 81% in Quantitative 
and Systems Biology). For the School 
of Natural Sciences, journal articles, 
citations, and number of grants were 
stronger. Books, grant dollars, and 
number of honors and awards were 
lower. That would be a reasonable 
pattern for a very young university with 
a disproportionately small number of 
full professors. As was true for other 
comparisons, the results are not 
prescriptive and, especially in this case, 
should not be used to establish a rigid 
individual faculty norm for evaluation. 
The norms are more meaningful at 
discipline and school levels. 


SUMMARY 


There are remarkably few published 
productivity standards in higher 
education (Chatman, 2016). Instead, 
analysis is typically parochial, treating 
history as a comparative standard, 

or, at the institutional level, treating 

a cluster of other universities as a 
comparative standard. The process of 


selecting peer institutions uses any of 
a variety of methods or combinations 
of predetermined or developed peer 
methods that have been well described 
elsewhere (Brinkman & Teeter, 1987) 
and continue to dominate higher 
education (the National Center for 
Education Statistics’ Executive Peer 
Tool, or ExPT). This is true even though 
much better data sources are available 
that support comparative analysis 

at the department level or at even 
smaller aggregates. This paper offers 

a constructed peer methodology as 
yielding a better, more-accurate, and 
more-valid peer because it accurately 
reflects the disciplinary composition 
of the home institution and isolates 
the comparison to the variable being 
considered. 


A constructed peer institution 

for comparison has important 
advantages to peers from traditional 
institutional methodologies. First, 

the process of constructing a peer 
produces comparative values at all 
levels of academic aggregation (e.g., 
department, school or college, and 
university). Second, the normative or 
standard values used to construct the 
peer can be tailored by department, 
school, or college so that each level 
can be based on its own tailored set 
of institutions. Perhaps the social 
sciences college and the engineering 
college of an engineering-focused 
university should have different 

peer sets. Third, the methodology is 
generalizable. The same steps used to 
construct a faculty salary peer can be 
used to produce a student satisfaction 
peer, an alumni engagement peer, a 
facility utilization peer, a development 
peer, etc. If a comparative measure 
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can be expressed at the level of a 
department and at a per capita rate 
common across institutions (e.g., 
faculty or FTE students) then the per 
capita rate can be inflated to reflect the 
home institution and support a direct 
comparison. For example, the mean 
level of satisfaction by disciplinary area 
for a comparable set of institutions 

can be weighted by local number of 
students by major and then compared 
at the college or institutional level. 
Fourth, in every case the constructed 
peer fits the home institution 
accurately. It has the same programs 

in the same relative and absolute 
amounts. For example, it has exactly 
the same number of faculty overall and 
by rank and discipline. It is a clone or 
doppelganger. Given that disciplinary 
differences are ubiquitous, institutional 
values used in comparison that 

ignore those differences might reflect 
disciplinary composition more than real 
differences. In other words, the home 
institution might appear to spend less 
on instruction per student because it 
is primarily a social sciences institution 
comprised of disciplines associated 
with less-expensive instruction. 
Likewise, student satisfaction and 
engagement varies by area of major 
(Chatman, 2010) and institutional 
comparisons of satisfaction or 
engagement will reflect disciplinary 
composition differences. Institutional 
measures that ignore differences 

in disciplinary composition (e.g., 
Voluntary System of Accountability™) 
can obscure real differences. Fifth, 

a variety of relative performance 
measures can be combined to yield a 
consistent dashboard or performance 
profile for departments, colleges, 

and the institution. For example, the 
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measures described in this paper 
produce an academic summary that 
includes cost per credit hour, faculty 
salaries, and faculty professional 
activities for a constructed peer that 
mirrors the home institution. 


A constructed peer also has two 
substantial disadvantages. First, it is 
more difficult to make transparent; 
also, in many cases, policies about 
sharing and reporting information 
among institutions prevent making 
the detail available. Second, it requires 
more effort on the part of the user 

to understand and the provider to 
describe because it is less familiar. It 

is more difficult to explain to higher 
education constituencies. A university 
president or chancellor will likely 
choose to report comparison to the 
average faculty salary for Pac-12 
institutions over the average faculty 
salary for a peer constructed from the 
bottom up using various combinations 
of Association of American Universities 
(AAU) public institutions. And, while 

it is less accurate and less valid, 
comparisons at the institutional 

level are often very similar to the 
constructed institutional average. 
Using an older sister university—for 
example, the overall faculty salary 
comparison to OSU's Faculty Salary 
Survey by Discipline—showed the 
sister university faculty salary average 
to be 9% higher. The comparison based 
on analysis using the constructed 
peer methodology by rank and 
discipline was 7% higher. If the only 
purpose of the peer comparison is to 
compare institutional-level values, 
then this method of peer construction 
is probably not worth the additional 
effort and loss of transparency. 


However, if the value of comparisons 
is extended to school and department 
levels, then constructed peers are 
preferable. If the methodology were 
to become more common, then its 
reporting would be less of a problem. 
We regularly use many summary 
measures and indices as if the meaning 
were simple and straightforward when 
they are actually remarkably complex. 
Some examples include the consumer 
price index, unemployment rate, Dow 
Jones industrial average, and even 
wind chill. 
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